Ultrasound imaging apparatus and method of controlling the same

ABSTRACT

An ultrasound imaging apparatus includes an image generator configured to scan an object to obtain an ultrasound image; a neural network configured to generate a virtual ultrasound image based on matching medical images of different modalities; an image converting part configured to convert a medical image of a different modality, previously obtained by scanning the object, into the virtual ultrasound image by using the neural network; and a matching part configured to determine a position of a virtual probe based on the ultrasound image and the virtual ultrasound image, and configured to match the ultrasound image with the medical image that corresponds to the determined position of the virtual probe.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2014-0144463, filed on Oct. 23, 2014, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND

1. Field

Apparatuses and methods consistent with exemplary embodiments relate toobtaining an ultrasound image from an object.

2. Description of the Related Art

An ultrasound imaging apparatus applies ultrasound to an object, detectsthe ultrasound, i.e. echoes, reflected from the object, and generates animage of a region of the object being examined, such as a cross sectionof soft tissue or a blood flow, based on the detected ultrasound toprovide information about the region being examined.

The ultrasound imaging apparatus has an advantage in that the ultrasoundimaging apparatus can obtain an image in real time. However, it may bedifficult to distinguish an outline, an internal structure, or lesion ofan organ in an ultrasound image because of noises included therein.

In recent years, a medical image obtained from another medical apparatusthat is matched with an ultrasound image is provided for the purpose ofperforming accurate diagnosis or delicate procedure on an object. Forexample, a magnetic resonance image that has relatively free scanningconditions and an excellent contrast and provides various images of softtissue or a computed tomogram having a higher resolution may be matchedwith an ultrasound image and provided.

SUMMARY

One or more exemplary embodiments provide an ultrasound imagingapparatus for matching a previously obtained medical image and anultrasound image obtained by scanning an object to provide the matchedimage and a method of controlling the same.

According to an aspect of an exemplary embodiment, there is provided anultrasound imaging apparatus, which includes: an image generatorconfigured to scan an object to obtain an ultrasound image; a neuralnetwork trained for generation of a virtual ultrasound image based onmatched images; an image converting part configured to convert a medicalimage previously obtained by scanning the object into the virtualultrasound image using the neural network; and a matching partconfigured to determine a position of a virtual probe to be applied tothe medical image based on the ultrasound image and the virtualultrasound image and to match the ultrasound image and the medical imagebased on the position of the virtual probe.

Here, the matching part may determine the position of the virtual probebased on an error between the ultrasound image and the virtualultrasound image.

Further, the image converting part may generate the virtual ultrasoundimage based on the position of the virtual probe.

Further, the image converting part may select a first region from themedical image based on the position of the virtual probe, input theselected first region into the neural network, and obtain an image of asecond region of the virtual ultrasound image.

Here, the first region may have a length determined based on a range ofultrasound in the object, and a width determined based on a resolutionof the ultrasound image.

Further, the ultrasound imaging apparatus may further include a learningpart configured to train the neural network using the ultrasound imageand the medical image that are matched at the matching part.

The medical image may include one of a magnetic resonance (MR) image, acomputed tomography (CT) image, a positron emission tomography (PET)image, and a single photon emission computed tomography (SPECT) image.

Further, when the matched medical image is input, the learning part maytrain the neural network such that the ultrasound image matched with themedical image is output.

In addition, the neural network may have a multilayer perceptronstructure.

According to an aspect of an exemplary embodiment, there is provided amethod of controlling an ultrasound imaging apparatus, which includes:scanning an object to obtain an ultrasound image; converting a medicalimage previously obtained by scanning the object into a virtualultrasound image using a neural network trained for generation of thevirtual ultrasound image based on matched images; and determining aposition of a virtual probe to be applied to the medical image based onthe ultrasound image and the virtual ultrasound image and matching theultrasound image and the medical image based on the position of thevirtual probe.

Here, the converting of the medical image may include converting themedical image into the virtual ultrasound image based on the position ofthe virtual probe set for the medial image.

Further, the matching of the ultrasound image and the medical image mayinclude resetting the position of the virtual probe based on an errorbetween the ultrasound image and the virtual ultrasound image.

Further, the matching of the ultrasound image and the medical image mayfurther include applying the neural network to the medical image andgenerating the virtual ultrasound image based on the reset position ofthe virtual probe.

In addition, the matching of the ultrasound image and the medical imagefurther may include matching the medial image and the ultrasound imagebased on the position of the virtual probe.

Further, the converting of the medical image may include: selecting afirst region from the medical image based on the position of the virtualprobe; and inputting the selected first region into the neural networkto obtain an image of a second region of the virtual ultrasound image.

Here, the first region may have a length determined based on a range ofultrasound in the object, and a width obtained based on a resolution ofthe ultrasound image.

In addition, the method may further include training the neural networkusing the medical image and the ultrasound image that are matched witheach other in the matching of the ultrasound image and the medicalimage.

As described above, the medical image and the ultrasound image arematched using the previously trained neural network, and thus themedical image and the ultrasound image can be more precisely matched.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will become more apparent by describingcertain exemplary embodiments with reference to the accompanyingdrawings:

FIG. 1 is a perspective view illustrating an ultrasound imagingapparatus according to an exemplary embodiment;

FIG. 2 is a control block diagram of the ultrasound imaging apparatusaccording to an exemplary embodiment;

FIG. 3 is a control block diagram for describing an image matcher of theultrasound imaging apparatus according to an exemplary embodiment;

FIG. 4 is a schematic view illustrating an example of a neural network;

FIG. 5 is a view for describing supervised learning of a neural network;

FIG. 6 is a flow chart for describing an example of a training method ofa neural network;

FIG. 7 is a view for describing a feature of an ultrasound image;

FIGS. 8A and 8B are views for describing an example of region selectionof a medical image and an ultrasound image;

FIGS. 9A and 9B are views for describing an example of region selectionof a medical image and an ultrasound image;

FIG. 10 is a view for describing generation of a virtual ultrasoundimage by using a neural network according to an exemplary embodiment;

FIG. 11 is a view for describing a virtual probe that is set for amedical image according to an exemplary embodiment;

FIG. 12 is a view for describing a virtual probe that is reset for amedical image according to an exemplary embodiment;

FIG. 13 is a control block diagram for describing an example of an imagematcher of an ultrasound imaging apparatus;

FIG. 14 is a flow chart for describing a method of controlling anultrasound imaging apparatus according to an exemplary embodiment;

FIG. 15 is a flow chart for describing determination of a position of avirtual probe according to an exemplary embodiment; and

FIG. 16 is a flow chart for describing a method of controlling anultrasound imaging apparatus according to an exemplary embodiment.

DETAILED DESCRIPTION

Certain exemplary embodiments are described in greater detail below withreference to the accompanying drawings.

In the following description, the same drawing reference numerals areused for the same elements even in different drawings. The mattersdefined in the description, such as detailed construction and elements,are provided to assist in a comprehensive understanding of exemplaryembodiments. Thus, it is apparent that exemplary embodiments can becarried out without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure exemplary embodiments with unnecessary detail

The terms used in the disclosure are selected as general terms usedcurrently as widely as possible upon consideration of the functions madein the disclosure, but they may be varied according to the intention orpractices of those of ordinary skill in the art, and the advent of newtechnology. Further, in specific case, terms arbitrarily selected by theapplicant are also used, and in this case, its meanings are mentioned incorresponding detailed description section, so the disclosure should beunderstood not by lexical meanings of the terms but by given meanings ofthe terms.

It will be understood that, when it is described that some parts“include” some components, this description does not exclude thepresence of other components throughout the specification, unlessotherwise described in particular. In addition, the terms “section,”“module,” “unit,” etc. used herein refer to the unit processing at leastone of a function and an operation, which can be realized by software,hardware such as a field programmable gate array (FPGA) or anapplication-specific integrated circuit (ASIC), or a combinationthereof. However, such components are not limited to the software or thehardware. Each of the components may be stored in an addressable storagemedium or may be configured so as to implement one or more processors.Accordingly, the constituent components may include, for example,software, object-oriented software, classes, tasks, processes,functions, attributes, procedures, sub-routines, and segments of programcodes, drivers, firmware, micro codes, circuits, data, database, datastructures, tables, arrays, variables or the like.

While such terms as “first,” “second,” etc., may be used to describevarious components, such components must not be limited to the aboveterms. The above terms are used only to distinguish one component fromanother. For example, a first component may be referred to as a secondcomponent without departing from the scope of the disclosure, andlikewise a second component may be referred to as a first component. Theterm “and/or” encompasses both combinations of multiple relevant itemsand any one of the multiple relevant items.

An ultrasound imaging apparatus according to an exemplary embodiment canscan an object to match the obtained ultrasound image with a medicalimage obtained from another modality of apparatus. For example, themedial image may be one of a magnetic resonance (MR) image, a computedtomography (CT) image, a positron emission tomography (PET) image, and asingle photon emission computed tomography (SPECT) image.

FIG. 1 is a perspective view illustrating an ultrasound imagingapparatus according to an exemplary embodiment.

As illustrated in FIG. 1, the ultrasound imaging apparatus 1 includes anultrasound probe 110, a main body 10, an operating panel 50, and adisplay 60.

The main body 10 may be provided with at least one female connector 45in the front lower side thereof. A male connector 40 provided at one endof a cable 30 may be physically coupled to the female connector 45. Theultrasound probe 110 and the main body 10 may be connected via the cable30.

The main body 10 may be provided with multiple casters 11 at a lowerportion thereof to secure mobility of the ultrasound imaging apparatus1. A user can fix the ultrasound imaging apparatus 1 at a specific placeor move the ultrasound imaging apparatus 1 in a specific direction usingthe multiple casters 11. In other words, the ultrasound imagingapparatus 1 may be a cart-type ultrasound imaging apparatus.

Alternatively, the ultrasound imaging apparatus 1 of FIG. 1 may be aportable ultrasound imaging apparatus that can be easily carried by auser. Here, the portable ultrasound imaging apparatus 1 may omit thecasters 11. The portable ultrasound imaging apparatus 1 may include, butnot limited to, a picture archiving and communications system (PACS)viewer, a smart phone, a lap-top computer, a personal digital assistant(PDA), a tablet personal computer (PC), or the like.

The ultrasound probe 110 is a portion that comes into contact with askin of the object, and transmits and/or receives ultrasound to and/orfrom the object ob. In detail, the ultrasound probe 110 generates theultrasound according to an input pulse, transmits the ultrasound to aninterior of the object ob, and receives reflected ultrasound, i.e.echoes, from a region of the interior of the object ob.

The operating panel 50 is a portion that receives instructionsassociated with operations of the ultrasound imaging apparatus 1. Theuser may input instructions for performing diagnosis start, diagnosisregion selection, diagnosis type selection, mode selection of a finallyoutput ultrasound image, etc. via the operating panel 50. Modes of theultrasound image may include an amplitude mode (A-mode), a brightnessmode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode),and a motion mode (M-mode) by way of example.

Further, the user may input instructions relevant to the image matchingvia the operating panel 50. For example, the user may input a virtualprobe position into the medical image and/or adjust the image matchingvia the operating panel 50.

In an exemplary embodiment, as illustrated in FIG. 1, the operatingpanel 50 may be located at an upper portion of the main body 10. Here,the operating panel 50 may include at least one of, for example, aswitch, a key, a wheel, a joystick, a track ball, and a knob.

The operating panel 50 may further include a sub-display 51. Thesub-display 51 is provided on one side of the operating panel 50, andmay display information associated with the operations of the ultrasoundimaging apparatus 1.

For example, the sub-display 51 may display a menu, a guideline, etc.for setting the ultrasound imaging apparatus 1 or adjust current settingof the ultrasound imaging apparatus 1.

Here, the sub-display 51 may be implemented as a touch panel. When thesub-display 51 is implemented as the touch panel, the user may touch thesub-display 51 to input control instructions.

The sub-display 51 is implemented as, for instance, a liquid crystaldisplay (LCD) panel, a light-emitting diode (LED) panel, or an organiclight-emitting diode (OLED) panel.

At least one holder 20 for the ultrasound probe 110 may be provided nearthe operating panel 50 to hold the ultrasound probe 110 in place. Thus,the user can hold the ultrasound probe 110 in the holder 20 of theultrasound probe 110 when the ultrasound imaging apparatus 1 is notused.

The display 60 may display the ultrasound images obtained during theultrasound diagnosis. As illustrated in FIG. 1, the display 60 may bemounted on the main body 10 integrally or separately.

Furthermore, the display 60 may include multiple first and seconddisplays 61 and 62 to display different images at the same time. Forexample, the first display 61 scans the object ob to display theobtained ultrasound medial image, and the second display 62 may displaya medical image that matches the obtained ultrasound medical image. Thefirst display 61 may scan the object ob to display the obtainedtwo-dimensional (2D) image, and the second display 62 may display athree-dimensional (3D) image.

Each of the displays 61 and 62 may employ a display such as a plasmadisplay panel (PDP), an LCD panel, an LED panel, an OLED panel, or anactive-matrix organic light-emitting diode (AMOLED) panel.

FIG. 2 is a control block diagram of the ultrasound imaging apparatus 1according to an exemplary embodiment. Referring to FIG. 2, theultrasound imaging apparatus 1 according to an exemplary embodimentincludes the ultrasound probe 110, a beamformer 120, an image generator130, a communicator 140, a storage 150, an image matcher 300, a maincontroller 160, the operating panel 50, and the display 60.

The communicator 140 is connected to another apparatus, and may transmitand/or receive data to and/or from the connected other apparatus. Thecommunicator 140 may be connected to another apparatus connected to anetwork 200, and may receive various data for performing the imagematching. For example, the communicator 140 may receive a neural network320 (see FIG. 3) for performing the image matching from anotherapparatus to be transmitted to the image matcher 300, or receive amedical image to be used for the image matching from another apparatus.

In detail, the communicator 140 may receive a trained the neural network320 from a learning apparatus 230. Here, the learning apparatus 230 isan apparatus for training the neural network 320, and may supervise andtrain the neural network 320 by using training data.

Further, the communicator 140 may receive the medical image obtained byscanning the object ob from a medical apparatus 210. In this way, themedical image received through the communicator 140 may be stored in thestorage 150 and used to generate a matched image.

Here, the medical apparatus 210 is designed to scan the object ob toobtain the medical image according to a preset method. The medicalapparatus 210 may be an apparatus having a modality different from thatof the ultrasound imaging apparatus 1. For example, the medicalapparatus 210 may be one of a magnetic resonance imaging (MRI)apparatus, a computed tomography (CT) apparatus, a positron emissiontomography (PET) apparatus, and a single photon emission computedtomography (SPECT) apparatus.

Further, the communicator 140 may receive information, such as diagnosisrecords, a treatment schedule, etc. of the object ob, which is stored ina medical server 220, and medical images obtained using the medicalapparatus 210, and/or transmit the ultrasound image obtained based onecho signals to the medical server 220.

Here, the medical server 220 administrates medical information that canbe used in diagnosis and treatment of the object ob. For example, themedical server 220 may administrate the information, such as diagnosisrecords, a treatment schedule, etc. of the object ob. Further, themedical server 220 may receive and/or administrate the medical imagefrom the medical apparatus 210, and transmit the stored medical image tothe ultrasound imaging apparatus 1 according to a request from theultrasound imaging apparatus 1.

Further, the communicator 140 may perform data communication withanother apparatus according to various wired and/or wirelesscommunication protocols, for example, according to the digital imagingand communications in medicine (DICOM) standard.

The ultrasound probe 110 comes into contact with the skin of the objectob, transmits the ultrasound to the object ob, and receives thereflected ultrasound, i.e. the echoes. To this end, the ultrasound probe110 may include a transducer. Here, the transducer T refers to a devicethat converts a given form of energy into another form of energy. Forexample, the transducer T may convert electrical energy into waveenergy, or vice versa.

In detail, the transducer T may include a piezoelectric material or apiezoelectric thin film. If an alternating current is applied to thepiezoelectric material or the piezoelectric thin film from an internalelectricity storage device such as a battery or an external powersupply, the piezoelectric material or the piezoelectric thin filmvibrates at a predetermined frequency, and ultrasound of a predeterminedfrequency is generated according to a vibration frequency.

On the other hand, when ultrasound of a predetermined frequency isreceived by the piezoelectric material or the piezoelectric thin film,the piezoelectric material or the piezoelectric thin film vibratesaccording to the frequency of the received ultrasound. Here, thepiezoelectric material or the piezoelectric thin film outputs analternating current of a frequency corresponding to the vibrationfrequency.

Further, various types of transducers such as a magnetostrictiveultrasonic transducer using a magnetostrictive effect of a magnet, apiezoelectric ultrasonic transducer using a piezoelectric effect of apiezoelectric material, and a capacitive micromachined ultrasonictransducer (cMUT) transceiving ultrasound using vibration of severalhundreds or thousands of micromachined thin films may be used as thetransducer T. In addition, different types of devices that can generateultrasound according to an electrical signal or generate an electricalsignal according to ultrasound may be used as the transducer T.

The beamformer 120 may apply a drive signal to the ultrasound probe 110,and/or beamform the echo signal received from the ultrasound probe 110.

To be specific, the beamformer 120 applies the drive signal to theultrasound probe 110. The beamformer 120 generates a drive pulse forforming transmission ultrasound according to a predetermined pulserepetition frequency (PRF), delays and outputs the drive pulse, andfocuses the ultrasound transmitted to the object ob.

Further, the beamformer 120 beamforms and outputs the echo signalreceived from the ultrasound probe 110. In detail, the beamformer 120may appropriately delay and focus the received echo signal based onreception directionality.

Further, the beamformer 120 may synthesize the delayed output echosignals. Here, the beamformer 120 synthesizes the multiple echo signalsto output the synthesized echo signal. The beamformer 120 may synthesizethe echo signals by applying a predetermined weighted value to the echosignals. The weighted value applied to the echo signals may bedetermined regardless of the echo signals, or determined based on theecho signals.

The image generator 130 generates an ultrasound image based on the echosignal output from the beamformer 120. For example, the image generator130 may generate at least one of an amplitude mode (A-mode) image, abrightness mode (B-mode) image, a Doppler mode (D-mode) image, anelastography mode (E-mode) image, and a motion mode (M-mode) image basedon the echo signal. Furthermore, the image generator 130 may generate a3D ultrasound image based on multiple ultrasound images obtained fromthe echo signals.

Here, the image generator 130 may correspond to one or more processors.The processor may be implemented as an array of numerous logic gates, ora combination of a general-purpose microprocessor and a memory in whicha program capable of being executed by the microprocessor is stored. Forexample, the image generator 130 may be implemented as a general-purposegraphic processing unit (GPU).

The storage 150 stores various pieces of information for driving theultrasound imaging apparatus 1. For example, the storage 150 may storeimage information about the diagnosis of the object ob, such as the echosignal and the ultrasound image, and store a program for driving theultrasound imaging apparatus 1.

Further, the storage 150 may store the medical image received via thecommunicator 140. Here, the medical image is obtained from the medicalapparatus 210 having a modality different from that of the ultrasoundimaging apparatus 1, and may be transmitted from the medical server 220or the medical apparatus 210 connected through the network 200.

Further, the storage 150 may include, but not limited to, a high-speedrandom access memory (RAM), a magnetic disc, a static random accessmemory (SRAM), a dynamic random access memory (DRAM), or a read-onlymemory (ROM).

Further, the storage 150 can be removably coupled with the ultrasoundimaging apparatus 1. For example, the storage 150 may include, but notlimited to, a compact flash (CF) card, a secure digital (SD) card, asmart media (SM) card, a multimedia (MM) card, or a memory stick.Further, the storage 150 may be provided outside the ultrasound imagingapparatus 1, and may transmit or receive data to or from the ultrasoundimaging apparatus 1 by wire or wireless.

The main controller 160 may control an overall operation of theultrasound imaging apparatus 1. In detail, the main controller 160controls each component to generate the ultrasound image of the objectob, and may match and display the generated ultrasound image and aprestored medical image.

Further, the main controller 160 controls the communicator 140 such thatthe medical image to be matched with the ultrasound image is received,and may store the medical image received via the communicator 140 in thestorage 150.

Further, the main controller 160 may correspond to one or moreprocessors. Here, the processor may be implemented as an array ofnumerous logic gates, or a combination of a general-purposemicroprocessor and a memory in which a program capable of being executedby the microprocessor is stored.

The image matcher 300 may match, under the control of the maincontroller 160, the ultrasound image obtained by scanning the object oband the medical image previously obtained by scanning the object ob.Hereinafter, the image matcher for matching the ultrasound image and themedical image will be described in detail.

FIG. 3 is a control block diagram for describing an example of an imagematcher of the ultrasound imaging apparatus.

As illustrated in FIG. 3, the image matcher 300 may include an imageconverting part 310 that converts a medical image into a virtualultrasound image, and a matching part 330 that matches an ultrasoundimage obtained by scanning the object ob and the virtual ultrasoundimage generated by the image converting part 310.

The image converting part 310 may convert the medical image into thevirtual ultrasound image using the previously trained neural network320. Hereinafter, prior to describing an operation of the imageconverting part 310, the neural network 320 will be described.

The neural network 320 is designed to engineeringly model a human brainstructure at which an efficient function of recognition occurs. Theneural network 320 may be implemented as hardware, software, or acombination thereof.

The human brain includes a basic unit of a nerve called a neuron. Eachneuron is connected to each other through a synapse and can processinformation in a nonlinear and/or parallel way.

The neural network 320 includes multiple units corresponding to theneurons, and the multiple units may be interconnected to each other witha predetermined connection strength. The neural network 320 that can beused to generate the virtual ultrasound image in this way has nolimitation. For example, the neural network 320 may be a convolutionalneural network.

FIG. 4 is a schematic view illustrating an example of the neural network320.

Referring to FIG. 4, the neural network 320 may include multiple layersL1, L2, L3, and L4 according to a multilayer perceptron structure.Namely, the multiple units U included in the neural network 320 may beclassified into the multiple layers L1 to L4.

The neural network 320 of the multilayer perceptron structure isimplemented with the multiple layers L1 to L4, and may be trained for amore complicated model.

Further, the units U classified into the multiple layers L1 to L4 may beinterconnected to each other with a predetermined connection strength.Here, each unit may be connected to only other units U having a highrelation.

Further, an output signal output from the output layer L4 of the neuralnetwork 320 may have a lower dimension than an input signal input viathe input layer L1. In other words, when M input signals are input tothe input layer L1, N output signals may be output from the output layerL4, wherein N is less than M.

FIG. 4 illustrates only an example of the neural network 320. The neuralnetwork 320 may include more layers, and a connection form of each unitmay vary. For example, each unit may be connected to only other unitsincluded in the neighboring layer according to a restricted Boltzmannmachine structure.

The human brain is trained by adjusting the connection form or strengthof the synapse. That is, the brain is trained by adjusting theconnection strength of the synapse in such a way that the brain weakensconnection between the neurons leading to an error of an incorrectanswer and reinforces the connection between the neurons leading to acorrect answer.

The neural network 320 is trained by imitating the aforementionedtraining way of the human brain. Here, the training refers to searchingand generalizing a pattern from predetermined training data. The neuralnetwork 320 is trained in such a manner that the connection between theneurons leading to the correct answer is reinforced. Hereinafter, anexample of the training way of the neural network 320 will be described.

FIG. 5 is a view for describing supervised learning of a neural network.

As illustrated in FIG. 5, the neural network 320 may be an object ofsupervised learning. The supervised learning is a method of training theneural network 320 using training data including input and output data.A correlation between the input and output data may be trained throughthe supervised learning.

In the training data of the neural network 320, an ultrasound image T2and a medial image T1 that are matched with each other may be used asthe training data for training the neural network 320. The medical imageT1 may serve as input data, and the ultrasound image T2 may serve asoutput data.

The neural network 320 is trained in such a manner that the virtualultrasound image output in response to the input medical image T1corresponds to the ultrasound image T2 that is the output data. Namely,the neural network 320 is trained to convert the medial image T1 intothe virtual ultrasound image by the training data.

A plurality of pieces of training data may be used for the training ofthe neural network 320. In this way, the neural network 320 may betrained to increase training precision the plurality of training data.For example, the training data may be large size data.

The supervised learning of the neural network 320 may be carried out bythe ultrasound imaging apparatus 1 according to an exemplary embodiment,however, the supervised learning of the neural network 320 may becarried out by the learning apparatus 230 connected to the network 200as described above.

Hereinafter, for convenience of the description, the neural network 320will be described to be trained by the learning apparatus 230 providedseparate from the ultrasound imaging apparatus 1. However, the neuralnetwork 320 may be trained by the ultrasound imaging apparatus 1.

FIG. 6 is a flow chart for describing an example of the training methodof the neural network 320.

Referring to FIG. 6, the learning apparatus 230 selects training datafrom a set of training data (S501). Since the plurality of pieces oftraining data may be used to train the neural network 320 as describedabove, the learning apparatus 230 may select the training data to beused for the training from among the set of the plurality of trainingdata. For example, the training data may include a medical image asinput data and an ultrasound image as output data.

The learning apparatus 230 applies the neural network 320 to the medicalimage of the training data and generates a virtual ultrasound image(S503). The learning apparatus 230 calculates an error between theultrasound image of the training data and the virtual ultrasound image(S505). If the calculated error is smaller than a reference value (‘Yes’to S507), the learning apparatus 230 terminates the training.

If the calculated error is greater than the reference value (‘No’ toS507), the learning apparatus 230 adjusts the connection strength of theneural network 320 based on the error (S509). In detail, the learningapparatus 230 may adjust the connection strength of the neural network320 according to an error backpropagation algorithm such that the errorbetween the virtual ultrasound image generated based on the input medialimage and the ultrasound image matched with the medical image isreduced.

In this way, the process of applying the neural network 320 to themedical image to generate the virtual ultrasound image, the process ofcalculating the error between the virtual ultrasound image and theultrasound image matched with the medical image, and the process ofupdating the connection strength based on the error are repetitivelyperformed, and thus the neural network 320 is trained.

Features of the ultrasound image may be considered for the training ofthe neural network 320. Hereinafter, the features of the ultrasoundimage will be described. FIG. 7 is a view for describing features of theultrasound image.

The ultrasound image is generated based on echo signals reflected fromthe object ob. In detail, brightness of each region of the ultrasoundimage is determined according to intensity of echoed ultrasound (or echosignals) reflected back from each region.

Referring to FIG. 7, for example, a 3D ultrasound image may be generatedbased on multiple 2-D ultrasound images corresponding first to n-thcross sections P1 to Pn. Here, brightness of a first point P11 on thefirst cross section P1 is determined according to intensity of echoedultrasound E reflected back from the first point P11.

Therefore, tissue located corresponding to the third cross section P3that is located at a relatively distant distance from the first pointP11 may be hardly affected by the echoed ultrasound E reflected backfrom the first point P11.

Tissue located near a travelling path of the echoed ultrasound Ereflected back from the first point P11 may influence the echoedultrasound E reflected back from the first point P11.

For example, when tissue located at a third point P13 reflects most ofultrasound received thereto, a magnitude of the echoed ultrasound may bereduced, and the echoed ultrasound E reflected back from the first pointP11 may be attenuated by the tissue located at the third point P13.

Further, the echoed ultrasound E reflected back from the first point P11may be attenuated or constructed by the echoed ultrasound reflected fromthe tissue located at the second or fourth point P12 or P14.

The ultrasound features as described above may be considered for thetraining of the neural network 320. In detail, the neural network 320may be trained by inputting only the region relevant to each region ofthe ultrasound image among the medial images. Hereinafter, regionselection of the training data will be described in detail.

FIGS. 8A and 8B are views for describing an example of the regionselection of the medical image and the ultrasound image.

Referring to FIGS. 8A and 8B, the learning apparatus 230 selects a firstregion from a medical image T11 and a second region from an ultrasoundimage T21. Here, the first and second regions correspond to each other.The neural network 320 may be trained through comparison of a virtualultrasound image generated by inputting the first region into the neuralnetwork 320 with the second region.

The first region selected from the medical image T11 and the secondregion selected from the ultrasound image T21 are correlated with eachother. That is, the first region may be selected based on a path ofultrasound transmitted to the second region selected from the ultrasoundimage T21 or a path of echo ultrasound reflected back from the secondregion.

Each of the first and second regions may include at least one pixel.Further, the first region may have a larger size than the second region.The first region may have more pixels than the second region.

The first and second regions may be constant in size. In detail, thefirst region may have a length determined based on the maximum range ofultrasound. The ultrasound transmitted from the ultrasound probe to theobject ob is gradually attenuated while progressing inside the objectob, and no ultrasound image T21 is obtained from tissue that is out ofthe maximum range of the ultrasound.

Therefore, information about a region that is not obtained from theultrasound image T21 is not needed for the training of the neuralnetwork 320, and thus the maximum length of the first region selectedfrom the medical image T11 may be shorter than the maximum range of theultrasound.

The first region may have a width determined according to a resolutionof the ultrasound image T21. For example, the higher the resolution ofthe ultrasound image T21 is, the narrower the width of the first regionis.

For example, when a region a of the ultrasound image T21 is selected asthe second region, the medical apparatus 210 may select a region A,which corresponds to a straight path between the second region and theultrasound probe, from the medical image T11 as the first region.

Further, when a region b of the ultrasound image T21 is selected as thesecond region, the medical apparatus 210 may select a region B, whichcorresponds to a straight path between the region b and the ultrasoundprobe, from the medical image T11 as the first region. Here, imageinformation about an ultrasound region that is not obtained within theregion B may be set to a preset reference value (e.g., zero).

In addition, when a region c of the ultrasound image T21 is selected asthe second region, the medical apparatus 210 may select a region C,which corresponds to a straight path between the region c and theultrasound probe, from the medical image T11 as the first region. Here,information about an ultrasound image region that is not obtained withinthe region C may be set to a preset reference value (e.g., zero.

In FIGS. 8A and 8B, each of the first and second regions has beendescribed to be in two dimensions, but may be set to three dimensions.FIGS. 9A and 9B are views for describing another example of the regionselection of the medical image and the ultrasound image.

As illustrated in FIGS. 9A and 9B, the learning apparatus 230 selects afirst 3D region from a 3D medical image T12 and a second 3D region froma 3D ultrasound image T22, and may train the neural network 320 torecognize the first region as an input of the neural network 320 and thesecond region as an output of the neural network 320.

The first region selected from the medical image T12 and the secondregion selected from the ultrasound image T22 are correlated with eachother. That is, the first region may be selected based on ultrasoundtransmitted to the second region selected from the ultrasound image T22or echo ultrasound reflected back from the second region.

Here, each of the first and second regions may include at least onevoxel. Further, the first region may have a larger size than the secondregion. The first region may have more voxels than the second region.

The first and second regions may be constant in size. In detail, aheight z of the first region may be determined based on the maximumrange of ultrasound. The ultrasound transmitted from the ultrasoundprobe to the object ob is gradually attenuated while progressing withinthe object ob, and no ultrasound image T21 is obtained from tissue thatis out of the maximum range of the ultrasound.

Therefore, information about a region that is not obtained from theultrasound image T22 is not needed for the training of the neuralnetwork 320, and thus the maximum height z of the first region selectedfrom the medical image T12 may be shorter than the maximum range of theultrasound.

A width x and a depth y of the first region may be determined accordingto a resolution of the ultrasound image T22. For example, the higher theresolution of the ultrasound image T22 is, the narrower the width and/ordepth of the first region is.

When a region d of the ultrasound image T22 is selected as the secondregion, the medical apparatus 210 may select a region D, whichcorresponds to a straight path between the second region d and theultrasound probe, from the medical image T12 as the first region.

FIG. 10 is a view for describing generation of the virtual ultrasoundimage by using the neural network 320 according to an exemplaryembodiment. FIG. 11 is a view for describing a virtual probe that is setfor the medical image according to an exemplary embodiment. FIG. 12 is aview for describing the virtual probe that is reset for the medicalimage according to an exemplary embodiment.

Referring to FIG. 10, the image converting part 310 of the image matcher300 applies the neural network 320 trained as described above to amedial image CT1, thereby converting the medical image into a virtualultrasound image corresponding to an ultrasound image US1. The imageconverting part 310 sets a virtual probe for the medical image and usesthe virtual probe as a reference in generating the virtual ultrasoundimage.

As illustrated in FIGS. 11 and 12, a portion of the medical image usedto generate the virtual ultrasound image varies according to a virtualprobe position, and thus the virtual probe needs to be set before thevirtual ultrasound image is generated. In detail, the virtual ultrasoundimage generated based on a first virtual probe position P1 and thevirtual ultrasound image generated based on a second virtual probeposition P2 or a third virtual probe position P3 may be different fromeach other.

Here, the position of the virtual probe may include coordinates at whichthe virtual probe is located and/or a direction of the virtual probe.

The position of the virtual probe may be determined by the matching part330.

The image converting part 310 generates the virtual ultrasound imagebased on the virtual probe position that is set for the medical image bythe matching part 330. The first virtual probe position P1 set by thematching part 330 may be designated by a user, or may be presetaccording to a predetermined protocol. Further, the first virtual probeposition P1 may be determined based on a position of the virtual probeused for previous image matching. Further, the first virtual probeposition P1 may be determined based on a position of the actualultrasound probe 110.

The image converting part 310 may generate the virtual ultrasound imageby repeating a process of selecting a first region from the medicalimage based on the virtual probe set for the medial image, a process ofinputting the selected first region into the previously trained neuralnetwork 320, and a process of obtaining an image of a second region ofthe virtual ultrasound image.

Here, a position of the second region of the virtual ultrasound image isdetermined according to a position of the selected first region. Thatis, the position of the first region and the position of the secondregion may have the same correspondence relation as the first region andthe second region in the training of the neural network 320.

For example, as illustrated in FIGS. 8A and 8B, when the position of theselected first region is the region A, the position of the second regionis the region a. When the position of the selected first region is theregion B, the position of the second region is the region b. When theposition of the selected first region is the region C, the position ofthe second region is the region c. Further, as illustrated in FIGS. 9Aand 9B, when the position of the selected first region is the region D,the position of the second region is the region d. In this way, theselected first region is input to the trained neural network 320, andthe virtual ultrasound image of the second region is obtained.

Further, the first and second regions selected when the virtualultrasound image is generated may respectively have the same sizes asthe first and second regions selected when the neural network 320 istrained.

The matching part 330 matches the medial image and the ultrasound imagebased on the virtual probe set for the medical image. The ultrasoundimage varies according to the position of the ultrasound probe 110. Forthe matching, a virtual probe corresponding to the actual ultrasoundprobe 110 may be set for a previously obtained medical image, and themedial image and the ultrasound image may be matched with each otherbased on the position of the virtual probe.

The matching part 330 may determine the position of the virtual probebased on the virtual ultrasound image generated by the image convertingpart 310 and the ultrasound image obtained by scanning the object ob.That is, the matching part 330 may reset the position of the virtualprobe for the medial image based on an error between the ultrasoundimage obtained by scanning the object ob and the virtual ultrasoundimage generated by the image converting part 310 such that theultrasound image obtained by scanning the object ob and the virtualultrasound image are matched with each other.

To be specific, the matching part 330 sets the virtual probe for themedical image. The matching part 330 calculates an error between theultrasound image obtained by scanning the object ob and the virtualultrasound image generated by the image converting part 310 based on theposition of the virtual probe set by the matching part 330.

If the calculated error is not greater than a preset reference, it maybe determined that the position of the virtual probe corresponds to theposition of the actual ultrasound probe 110. Thus, the ultrasound imageobtained by scanning the object ob and the medical image obtained fromanother medical apparatus 210 may be matched with each other based onthe position of the virtual probe set for the medical image.

If the ultrasound image and the medical image obtained from the othermedical apparatus 210 are matched with each other, the matching part 330may reset the position of the virtual probe according to movement theultrasound probe, update the medical image obtained from the othermedical apparatus 210, and continuously provide a matched image.

As illustrated in FIG. 12, when the position of the ultrasound probe 110moves from P2 to P3, the matching part 330 may change the position ofthe virtual probe from P2 to P3 in response to the movement of theultrasound probe 110, and continuously provide the matched imagecorresponding to the changed position of the virtual probe. If thecalculated error is greater than the preset reference, the matching part330 resets the position of the virtual probe set for the medial image.Here, the reset position of the virtual probe may be determinedaccording to the calculated error. For example, movement offset of thereset position may be set according to a degree of the error.Alternatively, the reset position of the virtual probe may be selectedby a user.

FIG. 13 is a control block diagram for describing another example of theimage matcher 300 of the ultrasound imaging apparatus 1.

An image matcher 300 according to another exemplary embodiment mayfurther include a learning part 340. As described above, the neuralnetwork 320 is trained using the medical image and the ultrasound imagethat are matched with each other. The learning part 340 may train theneural network 320 again using the medical image and the ultrasoundimage that are matched by the matching part 330.

Since the neural network 320 is trained again using the medical imageand the ultrasound image that are matched by the matching part 330, theneural network 320 can be trained in a more delicate way.

FIG. 14 is a flow chart for describing a method of controlling theultrasound imaging apparatus 1 according to an exemplary embodiment.Referring to FIG. 14, the ultrasound imaging apparatus 1 obtains amedical image (S601). Here, the medical image is obtained from the othermedical apparatus 210 before an ultrasound image of an object ob isobtained. For example, the medial image may be a CT image or an MR imageobtained by scanning the object.

The ultrasound imaging apparatus 1 applies the neural network 320 to themedical image based on a preset position of a virtual probe andgenerates a virtual ultrasound image (S605). Here, the preset positionof a virtual probe may be determined according to a certain ultrasoundscanning protocol or set by a user. Further, the preset position of thevirtual probe may be determined based on a position of the ultrasoundprobe corresponding to previous image matching. Further, the presetposition of the virtual probe may be determined based on a position ofthe actual ultrasound probe 110.

Here, the medical image is converted into the virtual ultrasound imagebased on the position of the virtual probe. In detail, the ultrasoundimaging apparatus 1 may convert the medical image into the virtualultrasound image by repeating a process of selecting a first region fromthe medical image based on the position of the virtual probe, a processof inputting the selected first region into the neural network 320, anda process of converting the selected first region into a second regionof the virtual ultrasound image.

The ultrasound imaging apparatus 1 scans the object ob to obtain anactual ultrasound image (S607). The actual ultrasound image may begenerated based on echo ultrasound reflected back from the object ob.

The ultrasound imaging apparatus 1 determines the position of thevirtual probe based on the virtual ultrasound image and the actualultrasound image (S609). As described above, the actual ultrasound imageis changed according to the position (e.g., coordinates and/ordirection) of the ultrasound probe 110. To match the medical image andthe ultrasound image, the virtual probe corresponding to the ultrasoundprobe 110 needs to be set for the medial image.

In this way, the ultrasound imaging apparatus 1 may determine theposition of the virtual probe based on the virtual ultrasound image andthe actual ultrasound image (S609). The determination of the position ofthe virtual probe will be described in detail with reference to FIG. 15.

The ultrasound imaging apparatus 1 matches the actual ultrasound imageand the medical image based on the determined position of the virtualprobe (S611). For example, a transform function determined based on theposition of the virtual probe may be applied to the medical image, and acoordinate system of the medical image may be substantially identical tothat of the ultrasound image.

As described above, the actual ultrasound image is changed according tomovement of the position of the ultrasound probe 110. The ultrasoundimaging apparatus 1 may update the position of the virtual probe appliedto the medical image according to movement of the position of theultrasound probe 110 and continuously provide the medical image matchedwith the actual ultrasound image.

FIG. 15 is a flow chart for describing operation S609 of FIG. 14.

Referring to FIG. 15, the ultrasound imaging apparatus 1 compares theactual ultrasound image with the virtual ultrasound image generated byapplying the neural network to the medial image based on the position ofthe virtual probe of operation S605 (S701). Here, there is no limitationto a method of comparing the actual ultrasound image with the virtualultrasound image. Further, a region of interest within the actualultrasound image may be compared with the virtual ultrasound image. Theultrasound imaging apparatus 1 determines whether an error between thetwo images is greater than a reference value (S703). Here, the referencevalue may be preset or selected by a user.

If it is determined that the error between the two images is greaterthan the reference value (‘Yes’ of S703), the ultrasound imagingapparatus 1 adjusts the position of the virtual probe (S705). Here, theadjustment of the position of the virtual probe may be determinedaccording to the calculated error. For example, movement offset of theposition of the virtual probe may be determined according to a degree ofthe error. Further, the position of the reset virtual probe may bedetermined by a user. When the position of the virtual probe is reset bythe user, the ultrasound imaging apparatus 1 may simultaneously displaythe actual ultrasound image and the virtual ultrasound image to allowthe user to easily adjust the position of the virtual probe.

The ultrasound imaging apparatus 1 applies the neural network to themedical image based on the adjusted position of the virtual probe toconvert the medical image into the virtual ultrasound image (S709).Here, the virtual ultrasound image may be generated by the same orsimilar method as in operation S605 of FIG. 14.

The ultrasound imaging apparatus 1 compares the actual ultrasound imagewith the virtual ultrasound image generated by applying the neuralnetwork to the medical image based on the adjusted position of thevirtual probe (S701). That is, the ultrasound imaging apparatus 1determines the position of the virtual probe by repeating operationsS701 to S709 until the error between the two images is equal to or lessthan the reference value.

FIG. 16 is a flow chart for describing a method of controlling theultrasound imaging apparatus according to another exemplary embodiment.

Referring to FIG. 16, the ultrasound imaging apparatus 1 obtains amedical image (S801). The ultrasound imaging apparatus 1 applies theneural network 320 to the medical image based on a preset position of avirtual probe and generates a virtual ultrasound image (S805).

The ultrasound imaging apparatus 1 scans an object ob to obtain anactual ultrasound image (S807). The actual ultrasound image may begenerated based on echo ultrasound reflected back from the object ob.

The ultrasound imaging apparatus 1 determines a position of a virtualprobe based on the virtual ultrasound image and the actual ultrasoundimage (S809).

In this way, the ultrasound imaging apparatus 1 may determine theposition of the virtual probe based on the virtual ultrasound image andthe actual ultrasound image (S809).

The ultrasound imaging apparatus 1 matches the actual ultrasound imageand the medical image based on the determined position of the virtualprobe (S811).

The ultrasound imaging apparatus 1 obtains training data from the actualultrasound image and the medical image that are matched with each other(S813) and update the neural network 320 based on the obtained trainingdata (S815). As described above, the training data of the neural network320 are the medical image and the ultrasound image that are matched witheach other. Therefore, in the ultrasound imaging apparatus 1, themedical image and the ultrasound image that are matched with each othercan be obtained as the training data.

The ultrasound imaging apparatus 1 updates the neural network 320 basedon the obtained training data. For example, the ultrasound imagingapparatus 1 may update the neural network 320 based on the training dataat a preset period, or the ultrasound imaging apparatus 1 may update theneural network 320 when the training data of a preset volume areobtained. Due to the update, the neural network 320 may be additionallytrained to provide a more precise virtual ultrasound image. Further, theneural network 320 may be customized to the user by the update.

The foregoing exemplary embodiments and advantages are merely exemplaryand are not to be construed as limiting. The present teaching can bereadily applied to other types of apparatuses. The description of theexemplary embodiments is intended to be illustrative, and not to limitthe scope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. An ultrasound imaging apparatus comprising: animage generator, implemented by at least one processor, configured toscan an object to obtain an ultrasound image; a matching part,implemented by the at least one processor, configured to set a positionof a virtual probe for a medical image, wherein the medical image isobtained from a medical apparatus having a modality different from thatof the ultrasound imaging apparatus; and an image converting part,implemented by the at least one processor, configured to convert themedical image into a virtual ultrasound image, based on the position ofthe virtual probe that is set for the medical image, using a neuralnetwork that is previously trained, wherein the neural network uses aplurality of ultrasound images and a plurality of medical images whichare matched to each other to learn a correlation between the pluralityof ultrasound images and the plurality of medical images, and is trainedto output a correlated ultrasound image when the medical image is inputas an input data, wherein the matching part is configured to match theultrasound image with the medical image that corresponds to the positionof the virtual probe and to reset the position of the virtual probebased on an error between the ultrasound image and the virtualultrasound image.
 2. The ultrasound imaging apparatus according to claim1, wherein the image converting part is configured to generate thevirtual ultrasound image based on the position of the virtual probe. 3.The ultrasound imaging apparatus according to claim 2, wherein the imageconverting part is configured to select a first region from the medicalimage based on the position of the virtual probe, to input the selectedfirst region into the neural network, and to obtain an image of a secondregion of the virtual ultrasound image.
 4. The ultrasound imagingapparatus according to claim 3, wherein the first region has a lengthdetermined based on a range of ultrasound transmission in the object. 5.The ultrasound imaging apparatus according to claim 3, wherein the firstregion has a width determined based on a resolution of the ultrasoundimage.
 6. The ultrasound imaging apparatus according to claim 1, furthercomprising: a learning part, implemented by the at least one processor,configured to train the neural network using the ultrasound image andthe medical image that are matched with each other by the matching part.7. The ultrasound imaging apparatus according to claim 6, wherein thelearning part is configured to train the neural network such that theultrasound image matched with the medical image is output as the virtualultrasound image in response to an input of the medical image.
 8. Theultrasound imaging apparatus according to claim 1, wherein the medicalimage includes at least one of a magnetic resonance (MR) image, acomputed tomography (CT) image, a positron emission tomography (PET)image, and a single photon emission computed tomography (SPECT) image.9. The ultrasound imaging apparatus according to claim 1, wherein theneural network has a multilayer perceptron structure.
 10. The ultrasoundimaging apparatus according to claim 1, further comprising: a displayconfigured to display the medical image that is matched with theultrasound image together with the ultrasound image.
 11. A method ofcontrolling an ultrasound imaging apparatus, the method comprising:scanning an object to obtain an ultrasound image; setting a position ofa virtual probe for a medical image, wherein the medical image isobtained from a medical apparatus having a modality different from thatof the ultrasound imaging apparatus; generating a virtual ultrasoundimage by converting the medical image into the virtual ultrasound image,based on the position of the virtual probe that is set for the medicalimage, using a neural network that is previously trained, wherein theneural network uses a plurality of ultrasound images and a plurality ofmedical images which are matched to each other to learn a correlationbetween the plurality of ultrasound images and the plurality of medicalimages, and is trained to output a correlated ultrasound image when themedical image is input as an input data; matching the ultrasound imagewith the medical image that corresponds to the position of the virtualprobe; and resetting the position of the virtual probe based on an errorbetween the ultrasound image and the virtual ultrasound image.
 12. Themethod according to claim 11, wherein the converting comprisesconverting the medical image into the virtual ultrasound image based onthe position of the virtual probe.
 13. The method according to claim 11,wherein the generating comprises generating a second virtual ultrasoundimage by applying the neural network to the medical image based on thereset position of the virtual probe.
 14. The method according to claim13, wherein the matching comprises matching the medical image and theultrasound image based on the reset position of the virtual probe inresponse to the error between the ultrasound image and the secondvirtual ultrasound image being equal to or less than a reference value.15. The method according to claim 13, wherein the converting comprises:selecting a first region from the medical image based on the position ofthe virtual probe; and inputting the selected first region into theneural network to obtain an image of a second region of the virtualultrasound image.
 16. The method according to claim 15, wherein thefirst region has a length determined based on a range of ultrasoundtransmission in the object, and a width determined based on a resolutionof the ultrasound image.
 17. The method according to claim 11, furthercomprising: training the neural network using the medical image and theultrasound image that are matched with each other.
 18. The methodaccording to claim 11, wherein the medical image comprises at least oneof a magnetic resonance (MR) image, a computed tomography (CT) image, apositron emission tomography (PET) image, and a single photon emissioncomputed tomography (SPECT) image.